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EXECUTIVE  SUMMARY 


This  report  views  the  current  status  of  the  Source 
Selection  process  as  it  is  being  applied  for  modifications 
effort  on  major  systems  within  the  Air  Force.  This  process, 
which  is  costly  in  terms  of  manhours  expended,  has  incurred 
the  wrath  of  many  Air  Force  managers  who  must  give  of  their 
top  personnel  to  this  time  consuming  process. 

Therefore,  this  writer,  currently  a contracting  of-' 
ficer  within  the  Air  Force  Logistics  Command,  who  recently 
participated  in  a Source  Selection  for  modifications  effort, 
felt  it  necessary  to  view  from  a lessons  learned  approach  the 
impact  of  this  technique  on  the  overall  procurement  process. 

Up  to  this  time  APLC  has  compiled  lessons  learned 
from  various  ALC's  who  have  conducted  Source  Selections  in 
accordance  with  Air  Force  Regulation  70-15.  It  is  from  these 
lessons  learned  that  the  nucleus  of  this  report  was  derived. 

Additionally,  a look  at  external  criticism  from 
government  review  committees,  as  well  as  industry,  is  included 
for  discussion. 

An  alternative  procurement  technique  is  provided  as 
a recommendation  to  those  who  believe  this  process  too  exten- 
sive or  otherwise  inappropriate  for  their  requirement. 

In  conclusion,  streamlining  and  modifying  the  process 
is  addressed  for  those  actions  considered  appropriate. 
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SECTION  I 


INTRODUCTION 

PURPOSE 


The  Source  Selection  process  is  an  evolutionary  pro- 
curement technique  developed  for  requirements  that  are  complex 
in  respect  to  contractor  achievement.  It  was  therefore  essen- 
tial that  contractors  be  carefully  selected  for  assurance  of 
meeting  the  Department  6f  Defense's  requirements.  Procedures 
for  doing  just  that  were  developed  and  DOD  directives  were  is- 
sued. Within  each,  service  regulations  were  written  to  adapt 
the  basic  DOD  directive  to  their  own  peculiar  programs.  It  is 
these  peculiar  programs,  which  do  not  fit  the  major  system  cat- 
egory, that  this  report  addresses. 

Specifically,  the  Air  Force  use  of  the  Source  Selec- 
tion process  for  major  modification  effort  to  existing  systems 
is  addressed. 

The  Air  Force's  two  major  commands,  AFLC  and  AFSC 
follow  AFR  70-15  procedures  for  this  type  effort.  AFSC  is 
primarily  involved  however  in  systems  acquisition.  AFLC,  on 
the  other  hand,  uses  Source  Selection  primarily  for  maintenance/ 
modification  of  existing  systems.  Therefore  this  command  (AFLC) 
through  the  cooperation  of  its  procurement  personnel  and  ALC 
contracting  officers  will  be  the  focal  point  for  lessons  learned. 

The  primary  purpose  of  viewing  the  Source  Selection 
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process  for  less  than  major  modifications  is  to  confront  the 
criticism  posed  by  its  users,  industry  and  governmental  com- 
mittees. Additionally,  revising  the  process  is  addressed, 
along  with  an  alternative  procurement  technique  proposal. 

GOAL 

The  goal  of  this  report  is  to  make  the  process  more 
understandable,  acceptable  and  updated  as  necessary  in  light  of  j 

lessons  learned  over  the  past  seven  years.  Additionally,  an  | 

overview  is  provided  as  to  the  usefulness  gf  the  source  selec-  i 

j 

tion  process  in  the  field,  its  long  suit  and  its  shortcomings.  j 

1 

This  topic  is  of  special  interest  to  this  writer,  a I 

i 

I 

contracting  officer  at  the  ALC  level,  who  recently  participated 
in  a Source  Selection  for  the  modification  of  an  existing  Air 
Force  system.  It  is  apparent  that  the  source  selection  process 
is  not  widely  accepted  within  the  Air  Force  for  any  and  all 
major  programs.  Both  industry  and  the  government  at  the  execu- 
tive level  have  criticized  the  Source  Selection  procedure. 

SCOPE 

This  process  is  viewed  in  the  setting  of  one  major 
command  (AFLC)  which  supports  the  various  systems  existing  with- 
in the  Air  Force.  This  view  will  have  applicability  to  the 
other  services  since  their  procedure  for  Source  Selection  are 
also  developed  from  the  primary  DOD  directive  4105.62  and  are 
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j similar  in  nature.  ! 

■ ^ 

, 

I Vi  ORGANIZATION 

Definitions  of  terms  and  explanation  of  acronyms 
^ used  herein  are  contained  in  Appendix  "A".  Appendix  "B" 

■f 

includes  regulations  that  concern  or  affect  the  source  selec-  j 

, tion  process  of  major  modification  effort.  1 

' 3 

Subsequent  sections  of  this  report  include  the  regu-  | 

■ 1 

lations,  procedural  concerns,  and  the  various  criticism  that  | 

has  arisen  from  the  users  viewpoint,  industry  and  the  Presi-  s 

dent's  Committee  of  Procurement.  In  addition  lessons  learned 

from  the  source  selections  conducted  over  the  last  seven  years 

within  AFLC  are  provided  in  Section  III. 

An  analysis  of  these  lessons  learned  is  provided 

in  Section  IV  and  is  summarized  in  Section  V. 

j 

Recommendations  for  the  Source  Selection  process, 
and  conclusions  are  provided  as  a wrap-up  to  this  report  in 
Section  VI. 
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SECTION  II 


PRESENT  SITUATION 
INTRODUCTION 


This  section  provides  a background  of  the  Source 
Selection  process  by  describing  the  regulations  and  directives 
followed  within  AFLC  and  AFSC,  the  commands  within  the  Air 
Force  which  primarily  use  the  process.  There  is  no  intent, 
however,  to  present  a "how  to"  approach  or  to  provide  detail  of 
the  process  itself.  This  is  accomplished  sufficiently  in  the 
regulations  and  in  an  unofficial  ASD  guide  entitled  The  Source 
Selection  Process  dated  1 Oct  1974.  Additionally,  Source  Sel- 
ection in  DOD;A  Comparison  of  Procedures  and  Techniques  by 
Andrew  E.  Turner  does  an  adequate  job  of  detailing  the  process. 

Procedural  concerns  at  the  AhC  level  will  be  described 
along  with  industry's  criticisms  and  the  Commission  on  Govern- 
ment Procurements  criticism  of  the  Source  Selection  process. 
These  concerns  focus  on  the  need  for  an  evaluation  of  the  pro-= 
cess  in  its  total  application. 

REGULATIONS , POLICY , GUIDANCE 

DODD4105.62  entitled  Selection  of  Contractual 
Sources  for  Major  Defense  Systems  is  the  basic  regulation  from 
which  all  other  Department  of  Defense  regulations,  policy, or 
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guidance  have  been  derived  concerning  the  Source  Selection 
process.  It  states  that  the  prime  objectives  of  the  process 
are  to  (a)  select  the  source  whose  proposal  has  the  highest 
degree  of  realism  and  credibility  and  whose  performance  is  ex- 
pected to  best  meet  Government  objectives  at  an  affordable  cost; 
(b)  assure  impartial,  equitable  and  comprehensive  evaluation  of 
competitors'  proposals  and  related  capabilities;  and(c)  maximize 
efficiency  and  minimize  complexity  of  solicitation,  evaluation, 
and  the  selection  decision  (9,2).^ 

This  regulation  established  Department  of  Defense 
policy  for  the  competitive  solicitation,  evaluation,  and  selec- 
tion of  contractual  sources  for  the  acquisition  of  major  Defense 
systems  in  accordance  with  the  Acquisition  of  Major  Defense 
System  directive  DODD  5000,1. 

DODD  4105.62  is  directed  at  the  major  defense  systems 
as  opposed  to  modifications  of  those  systems  or  sub-systems  6f 
which  this  report  is  directed.  From  this  directive  Air  Force 
Regulation  70-15  entitled  Source  Selection  Policy  and  Procedure 
was  derived.  This  regulation  prescribes  that  in  addition  to 
major  system  acquisitions  that  these  policies  and  procedures 

1 

This  notation  will  be  used  throughout  the  report  for 
sources  of  quotations  and  major  references.  The  first  number  is 
the  source  listed  in  the  bibliography.  The  second  number  is  the 
page  in  the  reference. 
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also  apply  to... "(3)  Any  modification,  maintenance,  services, 
or  other  program/project  estimated  to  require  $200  million  or 
more.  It  is  recognized  that  many  of  the  policies  and  proce- 
dures as  well  as  the  terminology,  contained  herein  apply  only 
to  major  defense  systems .. and ,.. b.  Policies  and  procedures 
contained  in  this  regulation  are  sufficiently  flexible  to 
accomodate  a wide  range  of  requirements.  They,  therefore,  may 
be  used  as  a guide  to  formally  evaluate  competitive  proposals 
and  to  select  sources  for  other  programs/projects  below  the 
dollar  threshold  prescribed  above.  In  this  context,  these  pro- 
cedures should  be  tailored  to  individual  program/project  re- 
quirements and  selectively  applied  to  avoid  excess  L\-e  costs." 


(7,1) 


AFR  70-15  is  the  principle  guidance  for  Source  Selec- 


tion within  the  Air  Force.  AFLC  Supplements  and  AFSC  manuals 
augment  this  regulation  for  specific  applicability  to  programs 
within  these  commands.  As  a response  to  various  Air  Logistic 
Center  personnel  voiced  concern  to  the  use  of  Source  Selection 
for  their  varied  requirements,  this  report  takes  a look  at 
problems  with,  alternatives  to  and  recommendations  for  its  use. 


PROCEDURAL  OBSERVATIONS 


The  use  of  Source  Selection  within  the  Air  Force  in- 


A k. 


r 


volves  both  major  systems  acquisitions  and  modification/main- 
tenance programs  that  affect  existing  systems.  The  latter  use 
of  Source  Selections  is  the  concern  of  this  report.  Consider- 
able application  of  Source  Selection  is  applied  by  both  AFLC 
and  AFSC  on  less  than  major  system  acquisitions  for  several 
reasons  : 

(1)  Source  Selection  provides  control  of  the  selection 
process  from  the  "top"  down. (This  will  be  defined  later) 

(2)  The  procedures  allow  flexibility  by  application  of 
judgement  concerning  cost,  schedule, and  performance. 

(3)  Cost  realism  can  be  carefully  examined. 

(4)  The  technical  proposals  received  allow  for  more  de- 
tailed evaluation  of  capability  than  normal  price  competition. 

Perhaps  there  are  other  reasons  for  the  extensive  use 
of  the  procedures,  but  these  are  considered  to  be  the  most  com- 
mon . 

A reiteration  of  the  objectives  of  Source  Selection 

i ... 

IS  appropriate  at  this  time.  They  are  (a)  select  the  source 

whose  proposals  has  the  highest  degree  of  credibility  and  whose 
performance  is  expected  to  best  meet  Government  objectives  at 
an  affordable  cost; (b)  assure  impartial,  equitable,  and  compre- 
hensive evaluation  of  competitors'  proposals  and  related  capa- 
bilities ; and  (c)  maximize  efficiency  and  minimize  complexity  of 
solicitation , evaluation  and  the  selection  decision. (9 , 2)  It  is 
the  third  objective  that  concerns  have  arisen. 
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From  an  ALC  point  of  view,  the  process  has  not  only 
lengthened  the  procurement  process  from  180-200  days  or  longer, 
but  has  also  complicated  the  solicitation  and  the  evaluation. 

Additionally,  AFLC  has  tracked  costs  of  conducting 
Source  Selections  and  it  would  appear  on  the  surface  that  the 
process  is  expensive.  Based  on  my  experience,  a typical 
Source  Selection  expends  in  excess  of  $100,000.00  of  Air  Force 
resources.  However,  these  costs  are  primarily  attributed  to 
wages  of  the  evaluation  personnel  involved.  These  costs  ex- 
clude the  effort  required  in  the  generation  of  the  requirement 
but  include  all  overhead  for  the  process  of  Source  Selection. 
Compared  to  a straight  competitive  negotiation  arena,  these 
costs  would  exceed  normal  competition  only  to  the  extent  that 
the  same  numbers  of  personnel  are  not  as  intensively  involved 
toward  successful  award  of  a contract. 

The  value  of  the  programs  involved  in  some  twenty  to 
thirty  Source  Selections  conducted  by  AFLC  and  AFSC  run  from  a 
few  million  dollars  to  several  hundred  million  dollars  on  pro- 
grams five  years  in  length.  The  cost,  therefore,  appears  insig- 
nificant in  regards  to  conducting  of  Source  Selection  in  view  of 
the  magnitude  of  the  programs  involved.  However  the  organize-’ 
tions  from  which  manpower  is  obtained  for  the  Source  Selections 
view  the  "costs"  not  in  dollars  al<'..'»e. 

The  standard  organization  for  Source  Selection  is 
pyramidal  as  shown  in  Figure  1(5,12)  and  uses  from  25  to  250 
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personnel . 


FIGURE  1. 

SOURCE  SELECTION  AUTHORITY/SOURCE  SELECTION  ADVISORY  COUNCIL 
SOURCE  SELECTION  EVALUATION  BOARD 

An  organization  used  by  AFSC  and  at  least  one  ALC  with- 
in AFLC  is  modified  to  some  extent  to  two  layers  as  shown  in 
Figure  2.  This  organization  does  not  necessarily  decrease  the 
numbers  of  personnel  required,  but  is  used  for  a simplification 
of  tasks.  This  will  be  addressed  further  in  Section  VI. 
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FIGURE  2. 

SOURCE  SELECTION  AUTHORITY 
SOURCE  SELECTION  COUNCIL/EVALUATION  GROUP 


Each  of  these  organizations  is  a "top"  down  approach. 
The  Source  Selection  Authority  goes  no  lower  than  a commander 
of  an  Air  Logistics  center  within  AFLC  but  has  varying  levels 
within  ASD  depending  on  potential  program  cost,  '^his  approach 
controls  the  source  selection  at  the  level  deemed  most  benefi- 
cial to  ASD. 

The  evaluation  team  on  a modified  organization  com- 
bines the  activities  of  the  Source  Selection  Advisory  Committee 
and  Source  Selection  Evaluation  Board  for  simplification  of 
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tasks  and  to  lessen  time  required  for  the  more  formal  approach. 

The  combined  organization  and  ASD's  leveling  of  the 
SSA  seems  to  best  serve  the  interests  of  the  service  in  assuring 
an  impartial,  sound  decision  (5,29)  where  source  selection  pro- 
cedures are  necessary. 

Procurement  planning  and  solicitation  preparation  are 
inherent  prerequisites  for  sound  source  selections.  These  two 
areas  are  critical  to  the  formal  Source  Selection.  The  evalu- 
ation criteria  included  in  the  Request  for  Proposal  must  be 
clear,  concise  and  complete.  When  judgement  is  to  be  applied 
in  evaluation  it  must  be  so  stated.  The  preparation  of  this 
criteria  is  time  consuming  and  must  be  so  in  relation  to  its 
criticality.  The  point  to  be  made  here  is  that  the  AFR  70-15 
procedures  require  a more  exacting  set  of  rules  to  be  followed 
for  evaluation  than  normal  competitive  proposals  where  cost  is 
the  primary  consideration. 

Even  though  the  solicitation  should  be  designed  so 
as  to  minimize  competitor  and  Government  expense  incident  to 
both  preparation  and  response  thereto  as  well  as  limiting  both 
the  solicitation  and  the  response  pages  (9,4)  the  source 
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in  several  areas.  These  include  some  that  were  just  described. 
Primarily,  the  volume  of  paper  required  in  response  is  consi- 
dered less  than  desirable  by  both  parties.  Other  criticisms 
include:  1)  Evaluators  not  well  trained;  2)  Underbidding  by 
contractors  frequent,  but  make  up  on  changes;  3)  Waste  inher- 
ent-too  many  contractors  bid-only  one  wins;  4)  boards  take 
safest  course  of  action; and, 5)  should  award  on  experience  on 
last  job.  (1,276) 

PRESIDENTAL  COMMITTEE  CRITICISM 

The  President's  Blue  Ribbon  Panel  of  1970  on  Federal 
Procurement  criticized  the  source  selection  procedure  in  the 
following  areas:  1)  too  time  consuming  and  too  complex;  2) scores 
of  competitors  tend  to  be  very  close;  3)  large  number  of  items 
in  each  proposal  levels  the  overall  ratings;  4)  numerical 
scores  obscure  more  important  issues,  and, 5)  process  wastes  time 
and  scarce  resources.  (4,3) 

The  above  criticisms  are  of  the  source  selection  pro- 
cess in  general.  It  would  appear  that  the  use  of  the  process 
on  less  than  major  systems  would  be  even  more  greatly  criticized. 
The  next  section  will  take  a look  up  close  and 'in  some  detail 
as  to  how  the  evaluator  views  the  process  on  these  lesser  pro- 
curements . 
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SECTION  III 


LESSONS  LEARNED 

INTRODUCTION 


This  section  provides  lessons  learned  on  ten  source 
selections  conducted  within  AFLC.  Therefore  the  problems  en- 
countered are  from  the  ALC  point  of  view.  The  purpose  of  this 
presentation  is  to  provide  both  the  advantages  and  disadvantages 
as  well  as  problems  encountered  with  the  APR  70-15  process  on 
less  than  major  programs.  It  may  appear  to  the  reader  that  the 
disadvantages  far  outweigh  the  advantages.  However  the  sum 
total  of  lessons  learned  is  placed  in  its  proper  perspective  in 
Section  IV  in  terms  of  analysis. 

The  lessons  learned  have  been  categorized  into  two 
areas:  Specific  and  miscellaneous.  Specific  lessons  learned 
require  some  elaboration  and  this  is  provided.  The  miscellaneous 
lessons  learned  are  listed  without  further  elaboration. 

SPECIFIC  LESSONS  LEARNED 

Lessons  learned  on  ten  source  Selections  conducted 
by  the  various  Air  Logistic  Commands  have  several  common  ingre- 


dients both  favoring  and  disfavoring  use  of  source  selection. 


The  team  approach,  uninterrupted,  in  the  review  of 
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the  Request  for  Proposal  was  unanimously  applauded.  This 
"Murder  Board"  approach  during  team  member  session  reduced 
RFP  error.  Also  in  connection  with  this  appraisal,  the  conti- 
nuous close  cooperation  between  the  various  functional  organ- 
izations during  Source  Selection  is  a favorable  key  element. 

The  free  flow  of  information  and  requirements  made  it  possible 
to  successfully  accomplish  these  large  dollar  Source  Selections. 

The  high  priority  afforded  Source  Selection  activities 
proved  to  be  another  key  to  success.  Perhaps  this  same  priority 
is  not  afforded  normal  competition. 

Several  areas  are  normally  evaluated  in  this  process 
for  major  modification  effort.  These  included  Management/Ex- 
perience, Quality/Safety,  Production/Facilities  as  well  as  cost. 
These  areas  require  personnel  experienced  in  the  specifics  be- 
ing evaluated  to  accomplish  the  most  efficient  job.  However, 
inevitably,  bias  is  involved  through  individual  background  and 
experience.  It  is  most  difficult  to  keep  this  bias  out  of  the 
evaluation  process. 

The  contract  def initization  group  who  analyze  the  cost 
and  prepares  and  distribute  the  negotiated  contract  must  have 
top  procurement,  policy,  legal,  pricing  and  clerical  expertise 
on  board.  The  first  problem  inherent  in  these  selections  of 
personnel  is  obtaining  people  of  this  calibre  and  having  them 
available  throughout  the  lengthy  process. 

14 
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Us§  of  this  type  cadre  dilutes  the  functional  organ- 
ization from  which  they  came.  Additionally,  the  organizations 
are  reluctant  to  provide  this  type  individual  for  any  length 
of  time. 

Additional  organizational  problems  exist  in  the  use 
of  high  grade  officials  who  chair  the  SSEB  and  SSAC  teams. 

Their  total  responsibilities  often  conflict  with  pressing  and 
problematical  source  selection  matters. 

Leaving  organizational  problems  and  looking  at  con- 
tractual matters,  the  preparation  of  the  Request  for  Proposal 
along  with  the  necessary  Definitions  and  Standards  is  very 
critical  and  complex.  The  rationale  for  selecting  evaluation 
area  precedence  and  the  weight  factors  assigned  requires 
tailoring  to  each  individual  requirement.  Additionally,  the 
definitions  and  standards  cannot  consider  items  in  the  evalu- 
ation process  if  such  topical  input  was  not  a requirement  of 
the  RFP.  Evaluation  criteria  must  be  adapted  precisely  to  the 
requirements  of  the  RFP. 

In  the  evaluation  process  much  time  is  lost  if  the 
evaluators  have  not  had  previous  Source  Selection  experience. 

It  is  necessary  to  use  some  type  of  familiarization  exercise 
if  the  team  is  to  function  effectively  and  timely.  This  too  is 
time  (manhours)  consuming. 

Since  each  source  selection  to  this  time  has  been 
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administratively  developed  on  an  individual  basis  within  AFLC 
each  ALC  develops  its  own  formatting  for  evaluation  purposes. 
Various  forms  are  required  for  consistency  of  evaluation 
throughout  the  source  selection  process. 

The  sensitivity  of  the  process  along  with  the  numbers 
involved  to  accomplish  the  effort  makes  the  security  required 
quite  difficult.  Discussions  concerning  a Source  Selection 
cannot  be  permitted  outside  the  confines  of  the  Source  Selection 
area.  Maintenance  of  files,  records,  forms,  reproduction  of 
necessary  paper,  all  require  sensitive  handling  and  consumes 
many  manhours.  Each  person  involved  in  a Source  Selection  must 
be  briefed  and  debriefed  by  written  certification  of  these 
security  requirements. 

Price  evaluation  is  also  quite  complex  with  respect 
to  the  interface  necessary  between  the  PCO  and  Price  analyst 
and  the  evaluating  teams.  Deficiencies  discovered  during 
evaluation  of  technical  aspects  of  the  proposals  must  be  coordi- 
nated between  the  CDG  and  the  evaluator.  It  is  critical  that 
all  matters  that  affect  price  be  ii  corporated  in  the  price 
analyst's  review. 

Contract  preparation  is  compounded  by  the  number  of 
proposals  evaluated.  This  is  necessary  for  timeliness  of  award 
once  decision  is  reached  by  the  SSA  and  to  insure  no  breach  of 
security  occurs  prior  to  award  that  might  invite  unwarranted 
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protest  actions.  Therefore,  additional  manhours  than  would 
normally  be  consumed  are  required  for  negotiation  of  contractual 
language  with  each  evaluated  offeror. 

The  scoting  process  used  during  source  selection  where 
contractor  inquiries  (for  clarification)  and  proposal  deficien- 
cies (for  correction)  are  involved  is  approached  differently 
from  ALC  to  ALC  and  between  AFLC  and  AFSC.  This  is  the  result 
of  directives  concerning  rescoring  of  proposals  following  sub- 
mission to  the  offeror  of  contractor  deficiencies.  Lack  of 
uniformity  and  purpose  in  this  area  invites  confusion. 

The  reports  of  the  SSEB  and  the  SSAC  that  follow  team 
evaluations  are  to  assess  total  risk  involved  between  proposals. 
The  combination  of  not  rescoring  proposals  and  assessment  at 
these  levels  allows  for  considerable  judgement  and  subjectivity. 

Depending  on  the  organization  affected  for  an  indivi- 
dual Source  Selection,  whether  three  tiered  or  two,  can  make 
considerable  difference  in  administrative  time  consumed.  This 
is  the  result  of  structuring  of  required  briefings  from  review 
levels  through  the  SSA.  If  many  offers  are  involved,  as  in  the 
case  of  some  engine  source  selections,  much  detail  is  required 
in  individual  briefings.  Not  only  in  preparation  of  briefings 
but  in  the  presenting  of  the  briefing  itself  to  the  SSA  are  many 
hours  consumed. 

The  pre-award  survey  process  requires  continued 
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security  and  consistency  in  surveying  each  potential  awardee. 

To  minimize  rumor  and  speculation  of  award  it  is  essential 
to  curb  unnecessary  conversation  between  the  survey  team  mem- 
bers and  contractor  personnel  during  the  survey.  This  effort 
complicates  the  evaluation/award  process. 

MISCELLANEOUS  LESSONS  LEARNED 

Miscellaneous  lessons  learned  include  the  following: 

1)  Part  time  evaluators  hinder  the  effectiveness  of  the  pro- 
cess . 

2)  Establishment  of  programs  and  procurement  responsibilities 
must  occur  early  (prior  to  establishing  SSA)  where  life  cycle 
costing  models  and  reliability  improvement  warranty  efforts  are 
included  in  the  procurement  process. 

3)  Higher  headquarters  informal  involvement  in  the  Source 
Selection  process  often  undermines  the  security  inherent  to 
Source  Selection. 

4)  Participation  by  joint  using  commands  is  required  in  the 
advanced  planning  stages  of  the  requirement  and  development  of 
RFP  if  delays  in  future  milestones  are  to  be  averted. 

5)  Manual  approval  on  contracts  by  the  SSA  is  essential  to 
curtail  award  information  prior  to  its  official  occurence. 

Review  at  higher  headquarters  degrades  the  sensitivity. 

6)  A composite  of  all  lessons  learned  on  previous  source 
selection  would  be  beneficial  to  those  source  selections  cur- 


*•-  rently  being  established. 

' 7)  Although  source  selections  can  be  accomplished  in  less 

than  200  days,  compressing  that  schedule  decreases  efficiency 

r 

i of  evaluation  and  negotiation. 

! 

I ■ OBSERVATIONS 

. As  has  been  demonstrated  by  the  lessons  learned 

‘ arrayed  in  this  section,  the  Source  Selection  Process  has  had 

\ some  growing  pains  yet  continues  to  endure.  The  question  that 

I currently  exists  is  whether  the  endurance  is  beneficial  to  the 

I 

[ overall  system  acquisition  process  as  it  pertains  to  modifica-' 

[ ... 

[ tions  to  existing  systems  currently  in  use.  Can  the  problems 

that  exist  with  the  system  be  eliminated  and  maintain  the  pro- 
cess? Is  the  process  necessary  at  all? 

The  next  sections  will  attempt  to  answer  these  ques- 
tions in  an  analysis  of  lessons  learned,  summary,  and  recommen- 
dations . 
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SECTION  IV 


ANALYSIS  OF  LESSONS  LEARNED 

RECAP  OF  SITUATION 

Competition  based  on  price  alone  for  procurements 
which  involve  complex  modifications  endangers  the  system  invol- 
ved of  a source  is  selected  who  cannot  meet  technical  perfor- 
mance. Evaluation  of  technical,  managerial,  and  other  aspects 
required  to  do  the  job  were  mdved  up  in  emphasis  over  price  to 
enhance  complete  performance  after  award.  Therefore,  Source 
Selection  procedures  were  structured  to  attempt  to  evaluate 
other  than  price  as  those  aspects  were  required  for  the  indivi- 
dual task.  The  nature  of  the  Source  Selection  process  re- 
quired some  subjectivity  regardless  of  the  attempts  at  minimi- 
zing the  same  through  a scoring  and  weighting  system. 

Additionally,  complexity  was  inherent  to  the  evalua-' 
tion  process  as  well  as  the  administrative  controls.  Additional 
manpower  and 'thus  expense  crept  into  the  procurement  costs.  It 
has  been  said  that  Source  Selection  is  too  costly  for  the  bene- 
fits. These  costs  are  not  just  the  monetary  expense  directly 
involved,  but  also  in  the  loss  of  resources  over  a lengthy 
period  of  time.  However,  evidence  does  not  sunoort  this  con- 
clusion, The  selection  of  one  source  based  on  price  alone  can 
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encounter  additional  modification  costs  that  could  pay  for  a 
dozen  source  selections.  Examples  of  this  can  be  cited  on 
both  engine  and  aircraft  modifications  of  recent  years. 

Concerning  subjectivity  inherent  to  the  process,  bias 
can  certainly  be  involved.  This  is  minimized  when  several 
people  evaluate  the  same  area  and  pool  their  individual  evalu- 
ations. However  external  subjectivity  is  difficult  to  control. 
"Political  and  socioeconomic  objectives  as  the  prevention  of 
local  and  regional  unemployment  and  the  fair  distribution  of 
defense  contracts  are  especially  susceptible  to  being  implemen- 
ted in  source  selection  decision.  Implementing  these  objectives 
often  conflict  with  other  source  selection  criteria  such  as 
rewarding  good  past  performance  and  choosing  the  firm  whose  pro- 
posal is  best  in  terms  of  the  technical  consideration ."  (2 , 379 ) 

But  evaluation  of  company  capability,  especially  in 
matters  concerning  technical  and  managerial  personnel,  are  neces- 
sarily subjective  and  this  creates  a special  set  of  problems. 

One  such  problem  is  the  difficulty  buying  agencies  have  in  ex- 
plaining to  losing  bidders  and  to  third  parties  (such  as  con- 
gressional investigating  committees)  the  elements  that  influ- 
enced the  final  choice.  On  the  other  hand,  the  Air  Force  in  its 
Source  Selection  management  of  competitions  has  kept  secret' the 
weight  assigned  to  various  criteria  and  the  reasons  for  specific 
decision.  This  practice  of  maintaining  secrecy  appears  typical 
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of  management  competition  of  other  services. 

The  conventional  arguement  against  secrecy  of  award 
criteria  in  source  selections  competition  emphasizes  the  oppor- 
tunity which  secrecy  affords  for  concealing  mistakes,  favoritism, 
or  worse.  However,  more  serious  is  the  possibility  that  firms 
will  not  perceive  where  they  stand  in  the  buying  agency's  esti- 
mation, and  therefore  will  not  be  incited  to  correct  capability 
deficiencies  and  improve  their  standing.  (2,372,3) 

SUBJECTIVITY  AND  COMPLEXITY 

As  can  be  seen  through  out  the  Source  Selection  pro- 
cess whether  a total  system  procurement  or  a modification  to 
that  system  three  elements  of  concern  exist.  Complexity  of  the 
process,  subjectivity  inherent  to  the  process  and  cost  of  pro- 
curement actions  are  inmeshed  in  large  or  small  transactions. 

The  size  of  the  organization,  the  control  of  the  func- 
tions within  the  organization  and  the  structure  of  the  organi- 
zation are  in  part  a summation  of  complexity  drivers.  Within 
the  structure,  complexity  is  compounded  by  superimposed  mile- 
stone events.  These  events  which  include  a Source  Selection 
Plan,  screening  of  sources,  standards  and  definitions,  RFP, con- 
ferences, evaluation,  negotiation,  briefings  and  ultimate  award 
must  be  tailored  in  such  a way  as  to  meet  requirements  and  yet 
allow  selection  based  on  sound  business  judgement. 
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The  administrative  problem  cited  in  "lessons  learned" 
are  the  result  of  the  nature  of  Source  Selection  itself,  but  as 
will  be  discussed  later,  need  not  be  of  primary  concern.  How- 
ever, complexity  of  tying  judgement  to  absolute  requirements 
has  compounded  administrative  problems. 

Subjectivity,  on  the  other  hand,  must  be  maintained 
in  whatever  procurement  method  is  used.  If  " sound  business 
judgement  " is  discarded  to  maintain  strict  adherence  to  price 
competition,  the  journalistic  proposals  often  received  will  be 
sufficiently  influential  to  contract  award  but  may,  as  often 
times  happened,  critically  damage  systems  support. 

COST 

Cost  of  the  procurement  process  of  source  selection 
is  directly  affected  by  manpower  used.  Intensive  evaluation 
procedures  can  require  many  people  for  long  periods  of  time. 
Staffing  of  an  interim  organization  and  maintaining  separate 
quarters  to  conduct  a Source  Selction  compound  costs  associated 
with  the  process.  These  costs  will  vary  with  procurement 
method  and  within  source  selection  procedure.  It  is  essential 
however  that  the  best  people  be  placed  in  the  evaluation  effort 
to  make  this  process  successful.  This  cost  to  a functional  or- 
ganization is  the  ultimate  sacrifice  and  more  expensive  than  the 
dollars  involved. 
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The  extent  to  which  the  three  variables  of  complex--' 
ity,  subjectivity,  and  cost  are  controlled  is  dependent  on  a 
case  by  case  evaluation  as  early  in  the  requirement  generation 
as  possible.  The  visibility  that  a given  requirment  takes  on 
also  dictates  the  structuring  of  the  process,  organization, 
com|)lexity  and  cost. 


SECTION  V 


SUMMARY 

In  suiiiinary,  seven  years  of  lessons  learned  on  Source 
Selections  has  highlighted  many  problems  with  the  process.  These 
problems,  many  of  which  are  administrative  in  nature,  have 
prompted  this  review  of  the  process  and  to  pointing  out  advan- 
tages and  disadvantages.  The  conclusion  and  recommendations  to 
follow  will  address  continued  viability  of  the  use  of  this  pro- 
cess and  alternative  methods  that  might  be  followed. 

The  problems  that  have  risen  during  the  use  of  Source 
Selection  procedures  are  a combination  of  broad  policy  outlines 
without  definitive  instruction  or  direction,  organizational 
complexity,  need  for  sensitivity  controls,  higher  headquarters 
involvement  at  the  micro  level,  lack  of  adequate  foresight  and 
planning  by  the  agency,  and  an  overall  lack  of  uniformity  of 
source  selection  procedures  administratively  in  one  major  com- 
mand. 

The  sum  total  of  all  of  these  problems  are  not  destruc- 
tive of  the  procurement  process.  However,  they  must  be  viewed 
by  each  command  for  planning  of  future  source  selections  if  in 
fact  it  is  desirable  to  maintain  the  process  for  given  require- 
ments. Correction  and/or  diminishing  many  of  these  problems 
has  been  undertaken  and  can  be  resolved  for  future  procurements. 
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The  problems,  even  though  they  are  troublesome,  do 
not  offset  the  primary  objective  of  the  use  of  formal  source 
selection.  The  selection  process,  because  of  subjectivity  in 
making  a "sound  business  judgment",  has  invited  criticism,  but 
has  withstood  inquiry  and  protest. 

In  conclusion  and  recommendation,  let  us  view  changes 
to  the  existing  process  that  might  be  beneficial  as  well  as 
alternative  procurement  methodology  that  may  be  more  effective. 


SECTION  VI 
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RECOMMENDAT I ONS/CONCLUS I ON 

RECOMMENDATION 

Various  "lessons  learned"  deal  with  administrative 
problems  that  primarily  affect  AFLC  procurement  using  Source 
Selection  procedures.  Adopting  ASD's  procedure  of  modifying 
the  organizational  structure  to  two  tiers  instead  of  three  will 
eliminate  or  minimize  several  problems.  The  most  obvious  pro- 
blem minimized  is  that  of  compounded  complexity  with  the  levels 
of  organization  eliminated  that  requires  additional  briefings 
and  reports.  Combining  the  SSAC  and  the  SSEB  into  one  working 
group  achieves  the  same  objectives  as  use  of  both  teams. 

An  additional  ASD  guideline  of  delegation  of  SSA  down 
through  the  various  levels  of  authority  based  on  dollar  value 
of  the  program  will  eliminate  many  Source  Selection  illS  of 
over  management.  AFLC  could  likewise  adopt  such  a criteria. 

The  PCO  at  a given  dollar  threshold  could  be  effective  as  SSA 
for  less  significant  programs. 

Establishing  within  each  agency  a singular  office  re- 
sponsible for  monitoring  and  coordinating  all  source  selections 
is  another  necessary  ingredient.  This  would  alleviate  growing 
pains  that  affect  each  and  every  Source  Selection  within  an  ALC 
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where  new  'players'  are  used  each  time.  Along  with  this  is 
the  necessity  to  maintain  corporate  memory  from  Source  Selec- 
tion to  Source  Selection  as  is  necessary  within  any  continuing 
organization. 

Maintenance  of  lessons  learned  over  the  long  haul  can 
be  accomplished  with  proper  dissemination  from  command  to  com- 
mand and  from  headquarters  to  each  agency.  Without  distribu- 
tion of  lessons  learned,  which  are  compiled  within  each  major 
command,  adequate  use  cannot  be  achieved. 

Guidance  in  the  form  of  regulations  and  directives 
must  either  be  explicit  to  the  Nth  degree  or  lessons  learned 
provided  each  new  Source  Selection  group  for  early  planning  and 
orientation.  The  AFSC  unofficial  document  previously  cited 
should  be  maintained,  updated  as  necessary,  and  adopted  offi- 
cially by  both  commands  for  use  as  training  and  orientation 
for  Source  Selection  activities. 

ALTERNATIVE 

The  process  can  and  does  accomf>lish  its  primary  ob- 
jective. However,  alternative  procurement  approaches  must  be 
viewed  for  feasibility  in  certain  instances.  Source  Selection 
because  of  its  nature  is  afforded  high  priority  within  each 
comxaand.  Therefore  strong,  tightly  woven  groups  are  pulled  to- 
gether often  at  the  expense  of  ongoing  organizations  from  which 
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the  personnel  are  taken.  A normal  two  step  or  modified  two 
step  negotiated  procurement  could  be  used  and  achieve  the  same 
objective  of  formal  Source  Selection  if  afforded  similar  pri- 
ority to  Source  Selection. 

A technical  proposal  with  all  the  ingredients  re- 
quired during  Source  Selection  could  be  evaluated,  negotiated, 
and  prices  obtained  subsequently  of  along  with  technical  pro- 
posal submissions.  The  difficulty  with  this  is  in  the  evalu- 
ation criteria  spelled  out  in  the  RFP  and  the  subsequent 
judgement  that  must  be  applied  to  such  an'  area  as  management 
or  technical  capability. 

However,  as  sound  business  judgement  is  achievable 
in  Source  Selection,  it  can  likewise 'be  achieved  in  normal  two 
step  or  modified  two  step  negotiated  procurements. 

The  advantage  to  this  procedure  over  that  of  source 
selection  would  be  in  the  manpower  required  by  limiting  the 
evaluation  to  a few  people  who  have  expertise  in  the  technical, 
managerial  and  financial  aspects  of  the  program. 

The  main  key  to  this  type  procedure  is  properly  struc- 
turing the  RFP  and  being  provided  the  authority  to  make  selec- 
tions based  upon  evaluation  criteria  clearly  spelled  out  at 
each  level  as  appropriate  for  the  dollars  estimated. 

Use  of  fewer  people,  less  reporting  required,  less 
micromanagement  from  upper  echelons  and  half  the  cost  are  some 
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V.’  of  the  advantages  for  this  most  flexible  procurement  technique. 

The  disadvantages  are  similar  to  those  in  Source 
Selection  and  they  include  subjectivity  inherent  to  judgemental 
decisions  and  the  necessity  to  maintain  sensitivity  during  the 
entire  negotiation  process. 

OBSERVATION 

Whether  or  not  either  technique  is  used,  it  is  im- 
portant that  each  offeror  be  candidly  apprised  of  his  standing 
or  appraisal  within  the  procuring  agency's  evaluation  following 
award.  Secrecy  which  is  currently  inherent  to  the  scoring  and 
weighting  system  must  be  eliminated  so  as  to  avoid  suspicion 
by  any  party. 

CONCLUSION 

In  conlusion  it  is  necessary  periodically  to  stand 
back  and  evaluate  as  objectively  as  possible  techniques  such 
as  Formal  Source  Selection.  A procedure  such  as  this,  with  its 
complexity  and  subjectivity  must  come  under  close  scrutiny  by 
the  public  to  insure  tax  dollars  are  wisely  spent.  Upgrading 
and  revising  our  Source  Selection  method  must  continue  with 
lessons  learned.  Therefore,  continuance  of  the  current  collec- 
tion of  lessons  learned  is  essential.  However  it  is  more  essen- 
tial that  these  lessons  learned  be  properly  fed  back  into  the 
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system. 


Openness  to  all  interested  parties  to  methods  of 
selection  must  be  expanded  to  eliminate  rumor  and  suspicion 
of  wrong  doing. 

Programs  such  as  modifications  to  major  systems,  even 
though  they  may  have  small  dollar  value,  necessarily  require 
contractors  who  can  technically  and  managerially  at  the  lowest 
overall  cost  to  the  government  accomplish  the  effort.  In  achie- 
ving this  goal.  Source  Selection  procedures  or  a two  step 
method  must  be  continued  with  adequate  attention  to  the  contrac- 
tor's overall  ability  to  achieve  the  objective. 
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ACRONYMS 


AFLC  Air  Force  Logistics  Command 

AFSC  Air  Force  Systems  Command 

ALC  Air  Logistics  Center 

ASD  Aeronautical  Systems  "Division 

CDG  Contract  Def initization  Group 

DOD  Department  of  Defense 

SSA  Source  Selection  Authority 

SSAC  Source  Selection  Advisory  Council 

SSC  Source  Selection  Committee 

SSEB  Source  Selection  Evaluation  Board 

SSAG  Source  Selection  Evaluation  Group 

RFP  Request  For  Proposal 


1 


DEFINITIONS 


Contracting  Officer-  Procuring  Contracting  Officer  (PCO)  j 

Contractor  Deficiencies-  response  sent  to  contractors  during  ] 

Source  Selection  Evaluation  citing  I 

discrepant  areas  in  the  proposal  I 

Contractor  Inquiries-  inquiries  sent  to  contractors  during  I 

Source  Selection  evaluation  for  clari-  | 

fication  of  an  area  in  the  proposal  I 

Definitions  and  Standards-  specific  criteria  by  which  pro-  j 

posals  are  scored  during  Source  | 

Selection  process  | 

Formal  Source  Selection-  lAW  AFR  70-15  and  synonymous  with  j 

"Source  Selection"  as  used  herein  i 

j 

Lessons  Learned-  term  used  to  denote  problems  encountered 

or  methods  used  which  were  either  successful 
or  unsuccessful  during  a Source  Selection 
process 

Murder  Board-  in  depth  review  of  a given  document  used  here- 
in for  review  of  a Request  for  Proposal 
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APPENDIX  B 


*REGULATI0NS7MANUALS/DIRECTIVES 


DOD  Directive  4105 . 62 , "Proposal  Evaluation  and  Source  Selec- 
tion", January  6,1976 


AF  Regulation  70-15,  "Source  Selection  Policy  and  Procedures" 
16  April  1976 


AFLC  Supplement  1,  AFR  70-15 , "Source  Selection  Policy", 
January  1977 


Armed  Services  Procurement  Regulation  (ASPR) - Indexed  by 
subject 


AFSCR  70-9,  "Source  Selection  Procedures",  16  August  1974 


AFSCR  80-15,  "Technical  Proposal  Evaluation  Procedures", 
11  April  1972 


* These  regulations  and  directives  are  only  those  that  have 
had  some  impact  on  this  report  and  are  not  intended  to  be 
totally  inclusive  of  all  that  affect  or  concern  the  Formal 
Source  Selection  Process. 
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4 . Report  of  the  Commission  on  Government  Procurements ,Vol . 1 . , 
Washington , D . C . ,1972 

5.  The  Source  Selection  Process,  a handbook  for  AFSC/ASD  per- 
sonnel involved  in  Source  Selection  Actions.  Wright-Patter- 
son  APB,  Ohio,  1974 

6.  Armed  Service  Procurement  Regulations,  Department  of  Defense, 
Washington,  D.C . , 1976 

7.  Air  Force  Regulation  70-15,  Source  Selection  Policy  and  Pro- 
cedures , Washington,  D.C.  1976 

8.  Air  Force  Logistics  Command  Supplement  1,  AFR  70-15,  Source 
Selection  Policy,  Wright  Patterson  AFB,  Ohio,  1977 

9.  DODD  4105.62,  Selection  of  Contractual  Sources  for  Major 
Defense  Systems,  Washington,  D.C.,1976 


